Method and program for providing feedback on surgical outcome

ABSTRACT

A method for providing a feedback on a surgical outcome by a computer includes dividing, by the computer, actual surgical data obtained in an actual surgical process into a plurality of detailed surgical operations to obtain actual surgical cue sheet data composed of the plurality of detailed surgical operations, obtaining, by the computer, reference cue sheet data about the actual surgery, and comparing, by the computer, the actual surgical cue sheet data with the reference cue sheet data, and providing, by the computer, the feedback based on the comparison result.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of International PatentApplication No. PCT/KR2018/010329, filed on Sep. 5, 2018, which is basedupon and claims the benefit of priority to Korean Patent Application No.10-2017-0182889, filed on Dec. 28, 2017. The disclosures of theabove-listed applications are hereby incorporated by reference herein intheir entirety.

BACKGROUND

Embodiments of the inventive concept described herein relate to a methodand a program for providing a feedback on a surgical outcome.

There is a need to develop a scheme capable of providing information toassist a surgeon in a surgical process. In order to provide theinformation to assist with the surgery, detailed surgical operations ofthe surgery must be recognized.

Conventionally, in order to design a scenario for optimizing thesurgical process, a medical image that is previously captured isreferred to or an advice from a highly skilled surgeon is referred.However, it is difficult to determine unnecessary processes only basedon the medical image, and the advice of the experienced surgeon is notcustomized to a specific patient.

Therefore, the medical image or the advice of the skilled surgeon maynot be utilized as auxiliary means for optimizing the surgical processfor a surgery target patient as a surgery target.

Accordingly, development of a method for minimizing unnecessaryprocesses in performing the surgery using a 3D medical image (forexample, virtual images of 3D surgical tool movements and internal organchanges caused by the movement of the tool) to optimize the surgicalprocess, and providing surgery assisting information based on theoptimized surgical process is required.

Further, recently, deep learning has been widely used for analysis ofmedical images. Deep learning is defined as a set of machine-learningalgorithms that attempt high-level abstractions (abstracting of keycontent or functions in a large amount of data or complex data) via acombination of several nonlinear transformation schemes. Deep learningmay be largely considered as a field of machine learning that teaches ahuman mindset to a computer.

SUMMARY

Embodiments of the inventive concept provide a method and a program forproviding a feedback on a surgical outcome.

Purposes that the inventive concept intends to achieve are not limitedto those mentioned above. Other purposes as not mentioned will beclearly understood by those skilled in the art from followingdescriptions.

According to an exemplary embodiment, a method for providing a feedbackon a surgical outcome by a computer includes dividing, by the computer,actual surgical data obtained in an actual surgical process into aplurality of detailed surgical operations to obtain actual surgical cuesheet data composed of the plurality of detailed surgical operations,obtaining, by the computer, reference cue sheet data about the actualsurgery, and comparing, by the computer, the actual surgical cue sheetdata with the reference cue sheet data, and providing, by the computer,the feedback based on the comparison result.

Further, the actual surgical data may be divided into the plurality ofdetailed surgical operations, based on at least one of a surgery targetportion, a type of surgical tool, a number of surgical tools, a positionof the surgical tool, an orientation of the surgical tool, and movementof the surgical tool included in the actual surgical data.

Further, at least one of a standardized name or standardized code datamay be assigned to each of the plurality of detailed surgicaloperations.

Further, the providing of the feedback may includes obtaining searchinformation used for searching for at least one of the plurality ofdetailed surgical operations, extracting at least one detailed surgicaloperation corresponding to the search information based on thestandardized name or the standardized code data, and providing afeedback on the extracted at least one detailed surgical operation.

Further, the reference cue sheet data may include optimized cue sheetdata about the actual surgery, or referenced virtual surgical cue sheetdata.

Further, the providing of the feedback may include comparing theplurality of detailed surgical operations included in the actualsurgical cue sheet data with a plurality of detailed surgical operationsincluded in the reference cue sheet data, and determining, based on thecomparison result, whether an unnecessary detailed surgical operation, amissing detailed surgical operation, or an incorrect detailed surgicaloperation is present in the actual surgical cue sheet data.

Further, the determining of whether the incorrect detailed surgicaloperation is present in the actual surgical cue sheet data may includecomparing movement of surgical tool corresponding to a detailed surgicaloperation included in the reference cue sheet data with movement ofsurgical tool corresponding to a detailed surgical operation included inthe actual surgical cue sheet data, and determining, based on thecomparison result, whether the detailed surgical operation included inthe actual surgical cue sheet data is incorrect.

Further, the method may further include adding the actual surgical cuesheet data to to-be-learned cue sheet data, obtaining a model forobtaining optimized cue sheet data using the to-be-learned cue sheetdata having the actual surgical cue sheet data added thereto, andperforming reinforcement learning on the obtained model.

Further, the method may further include detecting at least one surgicalerror situation from the obtained surgical information, and providing afeedback on the detected surgical error situation.

Further, the method may further include obtaining information aboutprognosis corresponding to each of one or more actual surgical cue sheetdata including the actual surgical cue sheet data, performingreinforcement learning based on the one or more actual surgical cuesheet data and the prognosis information thereof, and determining acorrelation between at least one detailed surgical operation included inthe one or more actual surgical cue sheet data and the prognosis, basedon the reinforcement learning result.

According to an exemplary embodiment, a computer program is stored in acomputer-readable storage medium, wherein the computer program isconfigured to perform the method as defined above in combination with acomputer as hardware.

BRIEF DESCRIPTION OF THE FIGURES

The above and other objects and features will become apparent from thefollowing description with reference to the following figures, whereinlike reference numerals refer to like parts throughout the variousfigures unless otherwise specified, and wherein:

FIG. 1 is a view showing a robot-based surgery system according to thedisclosed embodiment;

FIG. 2 is a flowchart showing a method for providing a feedback on asurgical outcome according to one embodiment;

FIG. 3 is a flowchart showing a method for calculating an optimized cuesheet data according to one embodiment; and

FIG. 4 is a flowchart showing a method for obtaining a feedbackaccording to one embodiment.

DETAILED DESCRIPTION

Advantages and features of the inventive concept, and methods ofachieving them will become apparent with reference to embodimentsdescribed below in detail in conjunction with the accompanying drawings.However, the inventive concept is not limited to the embodimentsdisclosed below, but may be implemented in various forms. The presentembodiments are provided to merely complete the disclosure of theinventive concept, and to inform merely fully those skilled in the artof the inventive concept of the scope of the inventive concept. Theinventive concept is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing theembodiments only and is not intended to limit the inventive concept. Asused herein, the singular forms “a” and “an” are intended to include theplural forms as well, unless the context clearly indicates otherwise. Itwill be further understood that the terms “comprises”, “comprising”,“includes”, and “including” when used in this specification, specify thepresence of the stated features, integers, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, operations, elements, components, and/orportions thereof. Like reference numerals refer to like elementsthroughout the disclosure. As used herein, the term “and/or” includesany and all combinations of one or more of the associated listed items.Although terms “first”, “second”, etc. are used to describe variouscomponents, it goes without saying that the components are not limitedby these terms. These terms are only used to distinguish one componentfrom another component. Therefore, it goes without saying that a firstcomponent as mentioned below may be a second component within atechnical idea of the inventive concept.

Unless otherwise defined, all terms including technical and scientificterms used herein have the same meaning as commonly understood by one ofordinary skill in the art to which this inventive concept belongs. Itwill be further understood that terms, such as those defined in commonlyused dictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art andwill not be interpreted in an idealized or overly formal sense unlessexpressly so defined herein.

A term “unit” or “module” used herein means software or a hardwarecomponent such as FPGA, or ASIC. The “unit” or “module” performs apredetermined role. However, the “unit” or “module” is not meant to belimited to the software or the hardware. The “unit” or “module” may beconfigured to reside in an addressable storage medium and may beconfigured to execute one or more processors. Thus, in an example, the“unit” or “module” includes components such as software components,object-oriented software components, class components and taskcomponents, processes, functions, attributes, procedures, subroutines,segments of a program code, drivers, firmware, a microcode, circuitry,data, database, data structures, tables, arrays and variables.Functionality provided within the components and the “units” or“modules” may be combined into a smaller number of components and“units” or “modules” or may be further divided into additionalcomponents and “units” or “modules”.

As used herein, a term “image” refers to multi-dimensional data composedof discrete image elements (e.g., pixels in a 2D image and voxels in a3D image). For example, the image may include a medical image of anobject obtained by a CT imaging apparatus.

As used herein, the term “object” may be a person or an animal, or aportion or an entirety of a person or animal. For example, the objectmay include at least one of organs such as a liver, a heart, a uterus, abrain, a breast, and an abdomen, and a blood vessel.

As used herein, a term “user” may be a surgeon, a nurse, a clinicalpathologist, a medical image expert, etc. as a medical expert, may be atechnician repairing a medical apparatus. However, the presentdisclosure is not limited thereto.

As used herein, a term “medical image data” refers to a medical imagethat is captured by a medical imaging apparatus, and includes allmedical images that may represent a body of an object in a 3D modellingmanner. The “medical image data” may include a computed tomography (CT)image, a magnetic resonance imaging (MRI) image, a positron emissiontomography (PET) image, etc.

As used herein, a term “virtual body model” refers to a model created ina conforming manner to an actual patient's body based on the medicalimage data. The “virtual body model” may be created by modeling themedical image data in a three dimensions manner or by correcting themodeled data to be adapted to an actual surgery situation.

As used herein, a term “virtual surgical data” means data includingrehearsal or simulation detailed surgical operation performed on thevirtual body model. The “virtual surgical data” may be image data ofrehearsal or simulation performed on the virtual body model in a virtualspace or may be data in which a surgical operation performed on thevirtual body model is recorded.

As used herein, a term “actual surgical data” refers to data obtained asa medical staff performs actual surgery. The “actual surgical data” maybe image data obtained by imaging a surgical target portion in an actualsurgical process, or may be data in which a surgical operation performedin the actual surgical process is recorded.

As used herein, a term “detailed surgical operation” means a minimumunit of a surgical operation as divided according to specific criteria.

As used herein, a term “cue sheet data” refers to data in which detailedsurgical operations into which a specific surgical process is dividedare recorded in order.

As used herein, a term “to-be-executed cue sheet data” refers to cuesheet data obtained based on the virtual surgical data obtained viasimulation by the user.

As used herein, a term “training virtual surgical cue sheet data” isincluded in the to-be-executed cue sheet data, and means cue sheet datacreated based on the virtual surgical data obtained via surgerysimulation by the user.

As used herein, a term “referenced virtual surgical cue sheet data”refers to cue sheet data about virtual surgery performed by a specificmedical person for construction of to-be-learned big data or surgeryprocess guidance.

As used herein, a term “optimized cue sheet data” means cue sheet dataabout an optimized surgical process in terms of a surgery time orprognosis.

As used herein, a term “to-be-learned cue sheet data” means cue sheetdata used for learning for optimized cue sheet data calculation.

As used herein, a term “surgery guide data” means data used as guideinformation during actual surgery.

As used herein, a term “computer” includes all of various devicescapable of performing computation and providing the computation resultto the user. For example, the computer may include not only a desktop(PC) and a notebook, but also a smart phone, a tablet PC, a cellularphone, a PCS (Personal Communication Service) phone, a mobile terminalof synchronous/asynchronous IMT-2000 (International MobileTelecommunication-2000), a palm personal computer (PC), and a personaldigital assistant (PDA). Further, when a head mounted display (HMD)apparatus includes a computing function, the HMD apparatus may be acomputer. Further, the computer may be a server that receives a requestfrom a client and performs information processing.

Hereinafter, embodiments of the inventive concept will be described indetail with reference to the accompanying drawings.

FIG. 1 is a view showing a robot-based surgery system according to adisclosed embodiment.

Referring to FIG. 1, a schematic diagram of the system capable ofperforming robot-based surgery according to the disclosed embodiment isillustrated.

According to FIG. 1, the robot-based surgery system includes a medicalimaging apparatus 10, a server 20, and a controller 30, an imaging unit36, a display 32, and a surgery robot 34 provided in an operating room.According to an embodiment, the medical imaging apparatus 10 may beomitted from the robot-based surgery system according to the disclosedembodiment.

In one embodiment, the robot-based surgery may be performed by the usercontrolling the surgery robot 34 using the controller 30. In oneembodiment, the robot-based surgery may be performed automatically bythe controller 30 without the user control.

The server 20 is a computing device including at least one processor anda communication unit.

The controller 30 includes a computing device including at least oneprocessor and a communication unit. In one embodiment, the controller 30includes hardware and software interfaces for controlling the surgeryrobot 34.

The imaging unit 36 includes at least one image sensor. That is, theimaging unit 36 includes at least one camera to image a surgery targetportion. In one embodiment, the imaging unit 36 is used in conjunctionwith the surgery robot 34. For example, the imaging unit 36 may includeat least one camera coupled with a surgery arm of the surgery robot 34.

In one embodiment, the image captured by the imaging unit 36 isdisplayed on the display 32.

The controller 30 receives information necessary for surgery from theserver 20 or creates information necessary for surgery and provides theinformation to the user. For example, the controller 30 displays thecreated or received information necessary for the surgery on the display32.

For example, the user controls movement of the surgery robot 34 bymanipulating the controller 30 while looking at the display 32 toperform robot-based surgery.

The server 20 creates information necessary for robot-based surgeryusing medical image data of the object (patient) previously imaged bythe medical imaging apparatus 10 and provides the created information tothe controller 30.

The controller 30 may display the information received from the server20 on the display 32 to present the information to the user, or may usethe information received from the server 20 to control the surgery robot34.

In one embodiment, imaging means that may be used in the medical imagingapparatus 10 is not limited particularly. For example, various medicalimaging means such as CT, X-Ray, PET, and MRI may be used.

Hereinafter, a method for providing a feedback on a surgical outcomewill be described in detail with reference to the drawings.

FIG. 2 is a flowchart showing a method for providing a feedback on asurgical outcome according to one embodiment.

Steps shown in FIG. 2 may be performed in time series by the server 20or the controller 30 shown in FIG. 1. Hereinafter, for convenience ofdescription, each step is described as being performed by a computer,but a subject to perform each step is not limited to a specific device.All or some of the steps may be performed by the server 20 or thecontroller 30.

Referring to FIG. 2, a method for providing a feedback on a surgicaloutcome according to an embodiment of the present disclosure includesdividing, by the computer, actual surgical data obtained from an actualsurgical process into a plurality of detailed surgical operations toobtain actual surgical cue sheet data composed of the plurality ofdetailed surgical operations (S200); obtaining, by the computer,reference cue sheet data about the actual surgery (S400); and comparing,by the computer, the actual surgical cue sheet data and the referencecue sheet data to provide a feedback based the comparison result (S600).Hereinafter, detailed description of each step will be described.

The computer may divide the actual surgical data obtained from theactual surgical process into the plurality of detailed surgicaloperations to obtain the actual surgical cue sheet data composed of theplurality of detailed surgical operations (S200). The computer createsthe actual surgical cue sheet data based on the surgical image imaged inthe surgical process by the surgery robot or based on data obtained in acontrol process of the surgery robot.

The detailed surgical operation constituting the cue sheet data refersto a minimum operation unit constituting the surgical process. Theactual surgical data may be divided into the detailed surgicaloperations based on several criteria. For example, the actual surgicaldata may be divided to the detailed surgical operations based on surgerytypes (e.g., laparoscopic surgery, robot-based surgery), anatomical bodyparts where surgery is performed, surgical tools as used, a number ofsurgical tools, an orientation or a position of the surgical tooldisplayed on a screen, movement of the surgical tool (for example,forward/reward movement), etc.

The division criteria and detailed categories included within thedivision criteria may be directly set by the medical staff learning theactual surgical data. The computer may perform supervised learning basedon the division criteria and the detailed categories set by the medicalstaff to divide the actual surgical data into the detailed surgicaloperations as a minimum operation unit.

Further, the division criteria and detailed categories included withinthe division criteria may be extracted via learning of a surgical imageby the computer. For example, the computer may calculate the divisioncriteria and detailed categories included within the division criteriavia deep learning (i.e., unsupervised learning) of actual surgical dataaccumulated as big data. Subsequently, the computer may divide theactual surgical data based on the division criteria created via thelearning of the actual surgical data to create the cue sheet data.

Further, in another embodiment, the actual surgical data may be dividedbased on a result of determining whether the actual surgical datasatisfies the division criterion via image recognition. That is, thecomputer may recognize the anatomical organ position on the screen, thenumber of surgical tools appearing on the screen, and the number of thesurgical tools on the screen within the image of the actual surgicaldata as the division criterion and may divide the actual surgical datainto the detailed surgical operation units based on the recognizeddivision criterion.

Further, in another embodiment, the computer may perform the divisionprocess for cue sheet data creation based on surgical tool movement dataincluded in the actual surgical data. The actual surgical data mayinclude various information input in a process of controlling thesurgery robot, such as the type and the number of surgical toolsselected by the user, information about the movement of each surgicaltool when the user performs robot-based surgery. Accordingly, thecomputer may perform division based on information included in theactual surgical data at each time point thereof.

Further, in one embodiment, the actual surgical data includes varioustypes of detailed surgical operation such as ablation and suture.Division is performed based on the division criteria. Specifically, aprocess of dividing the actual surgical data (for example, actualsurgery image) about the actual gastric cancer surgery into the detailedsurgical operations to create the cue sheet data is as follows.

For example, gastric cancer surgery includes a detailed surgicaloperation to ablate a portion or an entirety of a stomach containing atumor, and a detailed surgical operation to ablate a lymph node. Inaddition, various resections and connections are used depending on astate of the gastric cancer. In addition, each detailed surgicaloperation may be divided into a plurality of detailed surgicaloperations based on a specific location where the detailed surgicaloperation is taken and a direction of movement of the surgical tool.

For example, the detailed operation of the gastric cancer surgery may bedivided into an opening step, an ablation step, a connection step, and asuture step.

Further, a method of changing a disconnected state of an organ to aconnected state includes an in vitro anastomosis method of incising andconnecting at least 4 to 5 cm of an end of an anticardium, and an invivo anastomosis method in which about 3 cm of umbilicus is incised andincision and anastomosis occur in an abdominal cavity. Theabove-described connection stage may be divided in detailed sub-stepsaccording to the specific connection method as described above.

Furthermore, each surgery operation may be divided into a plurality ofdetailed surgical operations according to the position and the movementof the surgical tool.

Each of the divided detailed surgical operations has a standardized nameallocated thereto based on a location where the detailed surgicaloperation is performed and a pathway of the surgical tool.

In one embodiment, the standardized name may be variously defined. Forexample, when dealing with a specific portion as a lower right portionof the stomach, the name of the portion may be a name commonly used inthe medical field. More comprehensive or detailed names defined in thesystem according to the disclosed embodiment may be used.

Therefore, the surgery image about the actual surgery by the user may beorganized into information in a form of a cue sheet in which a pluralityof detailed surgical operations are sequentially arranged based on thestandardized names.

Further, in one embodiment, the cue sheet data may be created as codedata of specific digits based on the division criteria for dividing thesurgical data into the detailed surgical operations. That is, thecomputer divides the actual surgical data into standardized detailedsurgical operations by applying standardized division criteria anddesignating detailed categories within the division criteria. Thecomputer may allocate a standardized code value to each detailedcategory and allocate standardized code data to distinguish eachdetailed surgical operation.

The computer allocates, to each detailed surgical operation, digitalizedcode data obtained by allocating numbers or letters to categories inorder from a higher category to a lower category to which the specificdetailed surgical operations belong, according to the order ofapplication of the division criteria. Thus, the computer may create cuesheet data in a form in which not images of the divided detailedsurgical operations but the standardized code data of the detailedsurgical operations are listed. Further, the user may share or deliverthe actual surgical process by providing only the cue sheet datacomposed of the standardized code data.

In one embodiment, when the computer is a client terminal disposed inthe operating room or corresponds to the controller 30, the computer mayacquire the standardized code data from the surgery image, and transmitthe obtained code data to the server 20 such that the actual surgicalprocess is shared or delivered.

In one embodiment, when the computer corresponds to the server 20, thesurgical image is transmitted to the server 20. Then, the server 20 maycreate the cue sheet data and the code data.

Further, in one embodiment, the computer may allocate a standardizedname to a standardized code of each detailed surgical operation. Thus,the user may select and identify only a desired detailed surgicaloperation within the entire cue sheet. Further, in this case, the usermay easily grasp a progress of the surgery or the rehearsal by simplyviewing the cue sheet in which the detailed surgical operations aresequentially arranged based on the standardized names thereof, withoutviewing an entirety of the surgery image.

The cue sheet data may be converted into a surgical image using an imagedatabase for each detailed surgical operation. In the image data base,an image matching each code data may be stored. A plurality of imagesmatching each code data may be stored therein depending on a situation.For example, specific detailed code data may include different detailedsurgical operation images in the image database according to previouslyperformed operations.

Further, as each cue sheet data is matched with a specific virtual bodymodel and the matching result is stored, the computer may reproduce thecue sheet data as a surgical simulation image by sequentially applyingthe detailed surgical operations included in the cue sheet data to thevirtual body model.

Therefore, the image corresponding to the cue sheet data may bereproduced in the same point of view as that of the surgery image.Alternatively, the image corresponding to the cue sheet data may bereconstructed in a point of view different from that of the surgeryimage and the reconstructed image may be reproduced. Alternatively, theimage may be modeled in a 3D manner, and thus the viewpoint and aposition thereof may be adjusted according to the user's manipulation.

The computer acquires the reference cue sheet data about the actualsurgery (S400).

In one embodiment, the reference cue sheet data means optimized cuesheet data created by the computer.

In another embodiment, the reference cue sheet data refers to referencedvirtual surgical cue sheet data.

FIG. 3 is a flowchart showing a method for calculating the optimized cuesheet data according to one embodiment.

The computer acquires one or more to-be-learned cue sheet data (S420).The to-be-learned cue sheet data refers to learning target data that isto be learned for calculation of the optimized cue sheet data. Theto-be-learned cue sheet data may include cue sheet data created based onthe actual surgical data (i.e. actual surgical cue sheet data) or cuesheet data created based on the simulated actual surgical data forreference (i.e. referenced virtual surgical cue sheet data). The actualsurgical cue sheet data is created by the computer dividing the actualsurgical data according to the division criteria. The referenced virtualsurgical cue sheet data is not obtained in the user's surgery simulationprocess, but is created by performing simulation for purpose ofconstructing the learning target data or providing the same topractitioners for reference.

Thereafter, the computer performs reinforcement learning using theto-be-learned cue sheet data (S440). The reinforcement learning refersto one area of the machine learning, and refers to a method in which anagent defined in a certain environment recognizes a current state andselects a detailed surgical operation or a sequence of detailed surgicaloperations that maximizes reward among selectable detailed surgicaloperations or sequences thereof. The reinforcement learning may besummarized as learning of a scheme of maximizing compensation based onstate transition and compensation according to the state transition.

Then, the computer calculates the optimized cue sheet data using thereinforcement learning result (S460). The optimized cue sheet data iscalculated based on the shortest surgical time that may reduce thepatient's anesthesia time, a minimum bleeding amount, an essentialoperation group, and an essential performance sequence, etc. based onthe reinforcement learning result.

The essential operation group refers to a group of detailed surgicaloperations that must be performed together to perform a specificdetailed surgical operation. The essential performance sequence refersto a surgical operation sequence at which that operations must besequentially performed in a course of performing specific surgery. Forexample, surgical operations that must be performed sequentially and anorder thereof may be determined according to the type of surgery or thetype of the surgical operation.

Further, the computer may calculate situation-based optimized cue sheetdata based on the patient's physical condition, the surgery targetportion (e.g., tumor tissue) condition (e.g., a size, a location, etc.of the tumor) thereof the reinforcement learning. To this end, thecomputer utilizes the patient condition, the surgery target portioncondition, and the like along with the to-be-learned cue sheet data forlearning.

In one embodiment, the computer may perform virtual simulation surgeryon its own. For example, the computer may create a surgical processaccording to the type of surgery and the type of the patient based onthe disclosed surgical process optimizing method, and may perform thevirtual surgery simulation based on the created surgical process.

The computer evaluates results of the virtual surgery simulation. Thecomputer may perform the reinforcement learning based on virtualsurgical simulation information and evaluation information on the resultthereof, thereby to obtain an optimized surgical process.

A model trained to create the optimal surgical process may not create anoptimized surgical process according to an individual patient and a typeof surgery thereof because in actual surgery, body structures and typesof surgery of patients are different from each other.

Therefore, the computer may create a surgical process based on thepatient's body structure and the type of surgery thereof using thetrained model, and may perform virtual surgery simulation, based on thecreated surgical process. In this way, the computer may create anoptimized surgical process for an individual patient and a type ofsurgery thereof via the reinforcement learning.

The computer compares the actual surgical cue sheet data and thereference cue sheet data with each other and provides a feedback basedon a comparison result (S600).

In one embodiment, the operation of comparing the actual surgical cuesheet data and the reference cue sheet data with each other may beperformed by the computer or the controller 30 disposed in the operatingroom, or may be performed on the server 20.

When the comparison is performed by the server 20, the server 20acquires the reference cue sheet data, and compares the reference cuesheet data with the surgical image or the cue sheet data (code data)obtained from the controller 30.

When the comparison is performed by the computer or the controller 30placed in the operating room, the computer acquires the cue sheet datafrom the surgical image, and compares the cue sheet data with thereference cue sheet data received from the server 20.

In one embodiment, the feedback may be provided through a website or anapplication. For example, the feedback may be provided through anapplication installed in a mobile terminal of the surgeon. When thesurgery is finished, a notification related to the feedback may beprovided to the mobile terminal of the surgeon.

FIG. 4 is a flowchart showing a method for obtaining a feedbackaccording to one embodiment.

In one embodiment, the computer may compare a type and an order ofdetailed surgical operations included in the actual surgical cue sheetdata with a type and an order of detailed surgical operations includedin the reference cue sheet data, and may provide a feedback on thesurgical outcome based on the comparison result (S620).

For example, the computer may determine whether a missing detailedsurgical operation, an unnecessary detailed surgical operation, or anincorrect detailed surgical operation among detailed surgical operationsincluded in actual surgical cue sheet data, compared to the detailedsurgical operations included in the reference cue sheet data is present,and may provide a feedback on the surgical outcome, based ondetermination result.

For example, a required detailed surgical operation included in thereference cue sheet data may be omitted from the actual surgical cuesheet data.

Further, an unnecessary detailed surgical operation not included in thereference cue sheet data may have been included in the actual surgicalcue sheet data.

Further, a detailed surgical operation included in the reference cuesheet data may be included in the actual surgical cue sheet data, butdetails thereof may be modified or incorrect.

In this case, the type of the detailed surgical operation in thereference cue sheet data and the type of the detailed surgical operationin the actual surgical cue sheet data are identical with each other, butdetails thereof may be different from each other.

Thus, the computer determines whether each detailed surgical operationhas been performed correctly. Although a specific detailed surgicaloperation is to be included in an entire surgical process, the specificdetailed surgical operation may not be performed normally. For example,when the specific detailed surgical operation is an operation ofcatching a tissue at a specific location using tongs-shaped surgicaltool, the tool may catch the tissue at a deeper position than an optimaldepth or the tissue may be caught in a small area of the tongs such thatthe tissue is caught and then is removed from the tongs. This situationmay be determined as the incorrect detailed surgical operation. Thecomputer may recognize the tissue and the surgical tool in the actualsurgery image via image recognition to accurately recognize the detailedsurgical operation, and may compare the recognized detailed surgicaloperation with a correct detailed surgical operation in the optimizedcue sheet data for evaluation and feedback.

Further, the computer may analyze the actual surgical cue sheet dataitself, and provide information on a record of an abnormal situation oran unexpected situation based on the analysis result. For example, whena bleeding occurs in a specific site, the computer may provide at leastone of the fact that the bleeding has occurred, a location of thebleeding, or an amount of the bleeding.

Further, the computer may analyze records of the detailed surgicaloperation included in the actual surgical cue sheet data, may determinea cause of the bleeding based on the analysis result and may provide thecause.

Specifically, the computer analyzes a completed actual surgical cuesheet to search for a surgical error (S640). When a surgical errorsituation is detected according to a preset rule, the computer mayprovide the feedback indicating the surgical error.

For example, a general surgical error situation and a method ofproviding the feedback accordingly are as described below.

In an example, the computer may detect an event when a foreign objectremains in the patient's body, the computer may provide a feedbackindicating the event. According to a disclosed embodiment, the computerrecognizes all objects included in the surgical image, as well as alocation of the surgical tool, and the bleeding site, based on theobtained surgical image, and analyzes each of the objects. The computerdetermines a location, numbers, and an invasion time of the objectsincluded in the surgery image. Therefore, the computer may generate awarning when it is determined that the foreign substance introduced intothe surgery target portion has not been removed when the surgery iscompleted, and may provide a feedback that asks the user to check thesurgery target portion. In one embodiment, the computer may ask the userto check the surgery target portion even when the object introduced intothe surgery target portion is not identified in the image. For example,when it is not confirmed that the object that has invaded into thesurgery target portion is removed from the surgery target portion, theobject may not be included in the surgery image, but may remain at aninvisible site. Thus, the computer may provide the feedback to ask theuser to check the surgery target portion even when the object introducedinto the surgery target portion is not identified in the image.

In one example, the computer may detect a surgical operation event of awrong patient, and may provide a feedback indicating the event.According to a disclosed embodiment, the computer analyzes the surgeryimage in real time, and performs registration between organs in the 3Dmodeled image and actual organs. The computer tracks locations of thecamera and surgical tool in real time, and determines a surgerysituation and obtains information that a simulator follows in the actualsurgical process. In one embodiment, when the actual organ and the organin the 3D modeled image do not match with each other in a process ofperforming the registration between the patient's actual organ and theorgan in the 3D modeled image, the surgery may be performed on the wrongpatient. Thus, the computer may ask the user to check this situation.

In an example, the computer may detect a surgical operation event of awrong surgical target portion, and may provide a feedback indicating theevent. According to a disclosed embodiment, the computer determines asurgical situation, and provides a surgical guide image according to arehearsal result or an optimal surgical scheme. In this process, when aprocess of the actual surgery is different from the rehearsal result orthe optimal surgery scheme, the surgery may be performed on a wrong siteor a different type of surgery may have been performed. Thus, thecomputer may ask the user to check this situation.

In an example, the computer may recognize a situation of nerve damageand may provide a feedback indicating this situation. In a disclosedembodiment, when the actual surgery is different from the rehearsalprocess, as described above, the computer may provide a feedbackindicating this situation. When an important nerve or ganglion is cutdepending on a positional relationship between the patient's organ andthe surgical tool or when a risk that the surgical tool approaches thenerve is predicted, a warning may be provided to the user. Further, thecomputer visually presents a blood vessel, a nerve, and a ganglion thatare invisible in an overlapping manner with the surgical image usingimage registration and AR (augmented reality) or MR (mixed reality)technology. Thus, even after the surgery, the user may see an importantbody portion well, thereby helping to review the surgical process.

Further, in another embodiment, the method may further include addingthe actual surgical cue sheet data to the to-be-learned cue sheet dataused for calculation of the optimized cue sheet data, and performingreinforcement learning (S660).

In one embodiment, the actual surgical cue sheet data may be improvedcompared to the optimized cue sheet data. Therefore, the actual surgicalcue sheet data may be added to the to-be-learned cue sheet data and thenthe reinforcement learning thereof may be performed, thereby to obtain amodel capable of creating improved optimized cue sheet data.

Further, in another embodiment, the computer tracks prognosis of thepatient corresponding to each surgical record (S680). The computer mayperform machine learning using each surgery record and the prognosiscorresponding to each surgery record as to-be-learned data, therebydetermining a combination of surgical operations resulting in eachprognosis.

For example, the computer may analyze the surgical operations ofpatients with specific side effects, and thus the computer may derivedetailed surgical operations or combinations of detailed surgicaloperations that may cause even minor surgical side effects.

In one embodiment, the computer may acquire information about detailedsurgical operations that bring about each prognosis via thereinforcement learning. For example, the computer may perform thereinforcement learning based on operations performed in the surgicalprocess, and learned data about prognosis occurring when the operationsare included or are performed in a specific order. Based on thereinforcement learning result, the computer may determine what prognosis(i.e., what side effect) may occur due to a specific detailed surgicaloperation, continuous detailed surgical operations, or a combination ofdetailed surgical operations.

Further, the computer may output and provide the feedback to the user invarious ways. In one embodiment, the computer may extract detailedsurgical operations having defects and provide the extracted detailedsurgical operations as the feedback. For example, the computer mayextract and reproduce only images of the detailed surgical operationshaving the defects, thereby to help the user to grasp the defects.

Further, the computer may search for and provide detailed surgicaloperations included in the surgical process. For example, the surgery isusually performed for several hours or more, such that it is difficultfor the user to check an entire surgery image after the surgery forfeedback. Therefore, the computer may provide the cue sheet data. Whenthe user selects one or more detailed surgical operations included inthe cue sheet data, the computer may extract only the selected detailedsurgical operation and may provide the feedback related thereto. Forexample, the computer may extract and reproduce only images of theselected detailed surgical operations.

According to a disclosed embodiment, each detailed surgical operationhas a standardized name and standardized code data. Therefore, the usermay search for each detailed surgical operation based on thestandardized name or the standardized code data. The user may look atthe cue sheet data to easily check the progress of the surgery.

The method for providing the feedback on the surgical outcome accordingto one embodiment of the inventive concept as described above may beimplemented using a program (or application) to be executed incombination with a computer as hardware and stored in a medium.

The program may include codes coded in computer languages such as C,C++, JAVA, and machine language that a processor (CPU) of the computermay read through a device interface thereof, in order for the computerto read the program and execute methods implemented using the program.The code may include a functional code related to a function definingfunctions required to execute the methods, and an executionprocedure-related control code necessary for the processor of thecomputer to execute the functions in a predetermined procedure.Moreover, the code may further include a memory reference-related codeindicating a location (address) of an internal memory of the computer oran external memory thereto in which additional information or medianecessary for the processor to execute the functions is stored.Moreover, when the processor of the computer needs to communicate withany other remote computer or server to execute the functions, the codemay further include a communication-related code indicating how tocommunicate with any other remote computer or server using acommunication module of the computer, and indicating information ormedia to be transmitted and received during the communication.

The storage medium means a medium that stores data semi-permanently,rather than a medium for storing data for a short moment, such as aregister, a cache, or a memory, and that may be readable by a machine.Specifically, examples of the storage medium may include, but may not belimited to, ROM, RAM, CD-ROM, a magnetic tape, a floppy disk, and anoptical data storage device. That is, the program may be stored invarious recording media on various servers to which the computer mayaccess or on various recording media on the user's computer. Moreover,the medium may be distributed over a networked computer system so that acomputer readable code may be stored in a distributed scheme.

The steps of the method or the algorithm described in connection withthe embodiments of the inventive concept may be implemented directly inhardware, a software module executed by hardware, or a combinationthereof. The software modules may reside in random access memory (RAM),read only memory (ROM), erasable programmable ROM (EPROM), electricallyerasable programmable ROM (EEPROM), a flash memory, a hard disk, aremovable disk, CD-ROM, or any form of a computer readable recordingmedium well known in the art.

According to the disclosed embodiment, the feedback on the course andthe outcome of the surgery may be provided to the user, based on thecomparing result between the actual surgical process and the reference.

According to the disclosed embodiment, the feedback may be provided byextracting the necessary portion from the entire surgery image. Thus,even when the user does not look through the entire surgery image, theuser may check the necessary portion thereof.

The effects of the inventive concept are not limited to the effectsmentioned above. Other effects not mentioned will be clearly understoodby those skilled in the art from the above description.

While the inventive concept has been described with reference toexemplary embodiments, it will be apparent to those skilled in the artthat various changes and modifications may be made without departingfrom the spirit and scope of the inventive concept. Therefore, it shouldbe understood that the above embodiments are not limiting, butillustrative.

REFERENCE NUMERALS

-   -   10: Medical imaging apparatus    -   20: Server    -   30: Controller    -   32: Display    -   34: Surgery robot    -   36: Imaging unit

What is claimed is:
 1. A method for providing a feedback on a surgicaloutcome by a computer, the method comprising: dividing, by the computer,actual surgical data obtained in an actual surgical process into aplurality of detailed surgical operations to obtain actual surgical cuesheet data composed of the plurality of detailed surgical operations;obtaining, by the computer, reference cue sheet data about the actualsurgery; and comparing, by the computer, the actual surgical cue sheetdata with the reference cue sheet data, and providing, by the computer,the feedback based on the comparison result.
 2. The method of claim 1,wherein the actual surgical data is divided into the plurality ofdetailed surgical operations, based on at least one of a surgery targetportion, a type of surgical tool, a number of surgical tools, a positionof the surgical tool, an orientation of the surgical tool, and movementof the surgical tool included in the actual surgical data.
 3. The methodof claim 2, wherein at least one of a standardized name or standardizedcode data is assigned to each of the plurality of detailed surgicaloperations.
 4. The method of claim 3, wherein the providing of thefeedback includes: obtaining search information used for searching forat least one of the plurality of detailed surgical operations;extracting at least one detailed surgical operation corresponding to thesearch information based on the standardized name or the standardizedcode data; and providing a feedback on the extracted at least onedetailed surgical operation.
 5. The method of claim 1, wherein thereference cue sheet data includes optimized cue sheet data about theactual surgery or referenced virtual surgical cue sheet data.
 6. Themethod of claim 1, wherein the providing of the feedback includes:comparing the plurality of detailed surgical operations included in theactual surgical cue sheet data with a plurality of detailed surgicaloperations included in the reference cue sheet data; and determining,based on the comparison result, whether an unnecessary detailed surgicaloperation, a missing detailed surgical operation, or an incorrectdetailed surgical operation is present in the actual surgical cue sheetdata.
 7. The method of claim 6, wherein the determining of whether theincorrect detailed surgical operation is present in the actual surgicalcue sheet data includes: comparing movement of surgical toolcorresponding to a detailed surgical operation included in the referencecue sheet data with movement of surgical tool corresponding to adetailed surgical operation included in the actual surgical cue sheetdata; and determining, based on the comparison result, whether thedetailed surgical operation included in the actual surgical cue sheetdata is incorrect.
 8. The method of claim 1, wherein the method furthercomprises: adding the actual surgical cue sheet data to to-be-learnedcue sheet data; obtaining a model for obtaining optimized cue sheet datausing the to-be-learned cue sheet data having the actual surgical cuesheet data added thereto; and performing reinforcement learning on theobtained model.
 9. The method of claim 1, wherein the method furthercomprises: detecting at least one surgical error situation from theobtained surgical information; and providing a feedback on the detectedsurgical error situation.
 10. The method of claim 1, wherein the methodfurther comprises: obtaining information about prognosis correspondingto each of one or more actual surgical cue sheet data including theactual surgical cue sheet data; performing reinforcement learning basedon the one or more actual surgical cue sheet data and the prognosisinformation thereof; and determining a correlation between at least onedetailed surgical operation included in the one or more actual surgicalcue sheet data and the prognosis, based on the reinforcement learningresult.
 11. A computer program stored in a computer-readable storagemedium, wherein the computer program is configured to perform the methodof claim 1 in combination with a computer as hardware.